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This document descrites a modified Delphi technique 


ssential to the technique: 


for use in é¢etablishing research ode and priorities at the 


identificaticn of needs, collection 


f rankings of the relative 


importance of the identified needs by institutional administrators, 
calculation cf the rank of identified needs using an 
importance/consensus-séthod, feedback of rankings to campus 
,adainistrators, planningrof acticns with camspus administrators, and 
reporting of developed plans to all nigger none The modified 


during \ the firs 
more face-to-f 
and reportitg than ries the ;traditional Del 


oup interviews 


interaction during later rounds of data organizing 
ahs technique. It is 


suggested that this process is. both efficie t. and effective as a 
method for cbtaining institutignal consensus on research needs and 
priorities. Appended is a4 computational example for calculating the 
isportance/consensts rankings for use in the nodi fied Delphi process. 
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. A Modified-Delphi Technique for Obtaining Consensus on ‘July, 1977 
Institutional Research Priorities : 

“In many institutions decision-makers individually request research studies as 
they perceive a need for them. Reflecting "crisis oriented" management, such 
requests are often accompanied by imminent deadlines and ignore activitieg in 
progress. Trying to satisfy the needs of several administrators and the rigors 
of research methodology creates conflicting pressures on researchers. Accord- 
ing to Gubasta (1976), such conflicting pressures contribute to the diffusion 
of institutional research resources, diminished work quality, limited creativity 
and isolation .from institutional constituents. ‘ 
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Institutional decisongmaking processes also complicate the task of selecting 
and assigning priorities to institutional research projects. For example, 
Stake (1972) contends that effective decision-makers use unknown formulas 

and weightings to process a great deal of information in assigning priorities 
to objectives. Similarly, Reinow'g (1973) review of behavioral studies indi- 
cates that organizational decision-making is generally undisciplined and con- 
ducted without regard to systematic tools or techniques. 


seas aig: according to Gilmour (1976), differences in the décision-making 


pro@ésses used are 4 source of potential conflict between institutional decision- 
makers. and institutional researchers. Decision-makers typically use political 

or compromise ortented approaches and seek piecemeal or incremental solutions 
which are satisfactory. Researchers are generally apolitical, use rational- 
analytically oriented approaches and seek comprehensive Solutions which are 
optimal. 


Fortunately, several investigators have suggested aspects of a solution to the 
problem of selecting and assigning priorities to institutional research projects. 
Maier (1963) and Stake (1972), for example, contend tNat the availability of 
facts contributes to objective decision-making. Gubasta (1976) advocates meet- 
ing with key decision-makers to identify their needs, and Reinow (1973) argues - 
that an optimal decision-making procedure would include rational and extra- 
rational elements. 
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Similarly, Garner (1970) used a two step Survey process to identify institutiongl 
- research needs and priorities, First,he met with academic departments and, ad- 
ministrative units of his institution. In these meetings he administered an 
Inventory for Institutional Researgch which, among other areas of concern, solicited 
extensive suggestions for areas in which research was needed. From the results of 
the Inventory, 15 of the most prominent suggestions for research were'selected ’ 
for a second survey where rank-order Priorities for Institutional Research were 
established. 
In addition ‘to survey methods such as Garner's, the delphi technique has been 
used widely to establish a variety of priorities in education (c.g. Curran, 1972 
and Judd, 1972). Developed by the Rand Corporation, the delphi technique estab- 
lishes consensus among experts without requiring face to face meetings. Instead, 
participants complete a serics of questionnaires interspersed with controlled 
feedback (llelmer, 1966). Weaver (1971) ,ontends that "... although Delphi was 
originally intended as a forecasting tool, its more promising educational appli- 
cation scems to be ... (as) a planning tool which may aid in pooling prioritics 
held by members and constituencies of an organization (p,271)." . e 


A number of researchers have modified the original delphi technique in attempts 

to increase its efficiency or applicability. For example, in a study of priorities 
for research in education, Hughes (1975) used item-sampling procedures and gencr- 
aTizability theory analysis to rWMluce the time respondents spent completing 
questionnaires. In applying the technique, other investigators have reduced the 


number of iterations used. on 


vr 
A Modified Delphi Technique 


Stemming from a desire for a more systematic method for establishing research 

needs and priorities and based upon thig review of literature and an analysis of 
institutional decision-making processes, a modifiel delphi technique was designed 
and implemented. The six major steps included in this process are shown schemat- 
ically in Figyre 1. Fach step is described briefly in this section of the paper. 


a 
1. Identify Collect 
Needs * ; a. Rankings 


Feedback Plan 
Rankings : Actions 


Figure 1. Major Steps in a Delphi Technique for Research Consensus 
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1. Identify Needs. Institutional research.needs are identified through individ- 
ual interviews with administrators with institution-wide responsibilities 
(campus administrators), small group interviews with a campus administrator 
and those administrators directly responsible to him, or a combination of 
these. These interviews should focus on identifying major unmet, institu- 
tional and organizational unit needs for which institutional research may 
be useful. To minimize the development of a lengthy "Christmas wish" list, 
it may be’ necessary to limit the input of organizatioral unit administrators 
to their single, mogt pressing need. 

2. Collect Rankings. A composite list is prepared of the research'needs identi- . 
fied in the interviews. Next, all institutional administrators are asked, 
from their perspective, to rank the list of needs according to cgmparative 
institutional need. To make the ranking task manageable, the list of needs 
should not exceed 15 - 20 items. 


Rank Needs by Impvortance/Consensus. The importance/consensus method (Raratta, 
1974) is used to rank the research needS. Considering both the importance 
assigned by a group of respondents and the degree of consensus across respon- 
‘dents, the importance/consensus method is superior to either importance or 
consensus ranking methods for assigning research priorities. 


The six general steps in ranking needs by importance/consencus include: 


+ First, calculate the importance rating for each need. The importance 
_ rating is the mean of responses for each need. 


* Second, calculate the consensus score for each need. Consensus is 
measured by’ Uhl's (19.71) convergence score, which is the sum of the 
absolute value of each respondent's choice minus the importance rating 
for each need. , 


* Third, rank the importance ratings from the most to the least important,- 
assipning a rank of one to the need with the smallest mean ratings. 
. : ~ 

* Fourth, rank the consensus scores from greatest to least consensus, 
assigning a rank of one to the need with the smallest consensus score. 


+ Fifth, sum importance and consensus ranks for each need. 


+ Sixth, re-rank the sum of ranks, assigning a rank of one to the item 
with the smallest sum of ranks. 


A computafioral examle of these steps for ranking needs by, importance/consensus 
is included in the Appendix. . 


Importance/consensus ranking can be calculated for all administrators or for 
appropriate organizational sub-groups. 


4. Feed Rack Rankings to Campus Administrators. A summaty table listing the 
research needs by importance/consensus can be distribdted to*administrators 
with campus wide responsibilities. The summary table may have a colum 
heading for "Action Plan."’ . 
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Plan Actions with Campus Administrators. In this step, the researcher meets 


with the group of campus Sno SLAC ONS: to discuss actions to be taken on 
each research’ need. : 
A predetermined list of action choices may facilitate group decision-making. . 
Action choices may include: research planning or conducting a study, research 
Supporting a study conducted by « by one or more organizational units, employ ing a 
consultant to plan or conduct: a study, or sxiesoing a study for later review. 


If poesibie, the researcher should lead the’ discussion in which an ‘action choice : 
is made for each research need. ranked, ‘ = 
Report. Plans to All hiubspepators, After an action entice has been made for 
each research necd, the researcher should report these plans to all adminis- 
trators involved in the study. A two column summary table may be sufficient 
to communicate established priorities and action plans to al administrators 
who might request studies. The first column of the summary table would be 
a ranking of the research needs by importance/consensus. THe second coluna 
would show the .dction’ planned for each need listed. Ilaving participated in 
planning the actions, campus administrators can provide decision-making 

. rationales to those administrators responsible to them. 
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A Comparison of Traditional and Modified Delphi’ Techniques 


Table 1 presents a general comparisdn. of this.modified delphi with the traditional 
delphi technique. The step by step conparis on which follows includes the ratio- 
nale for aach Rod SeeE Aan 


L - TABLE 1 


A Comparison of Traditional 
and 
Modified Delphi Techniques 


Round | Traditional t pedintid | Modified Delphi ' 
- See eee 


Open-ended guidsktonnaizs cu Individual or small grovp interviews 
Instrument to all j Instrument tovall 
Instrument + Feedback to all Feedback to campus administrators 


Instrument + Feedback + Plan actions with campus adminis~ 
Justification of divergent _trators 
‘views to all 


’ e 
Final report to all Final report £0 all 


rE ee == 
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Interviews, rather than an open-ended questionnaire, are used to identify re- _ 
search needs‘ in the first step of the modified technique. In contrast to a 
questionnaire, an interview permits immediate idea clarification and can focus 
‘on "need to know" rather than "nice to know" concerns. In addition, by aggre- 
gating several small, related projects into one longer one, or vice versa, inter- 
views facilitate a consistent sete) Bee Renan Sy. aeROes expressed needs. 


A face-to-face approach to identifying needs is advocated by Garner (1970) and by 
Gubasta (1976). Furthermore, this approach is consistent with Maier's principle 
that the idea~getting process should be separated from the idea-evaluation 
process. (1963, p. 247). 3 


Following the collectfon and analysis of individual responses from all participants, 
written results are fed-back to participants. At this stage the modified delpht 
differs from the eager paid) alee feedback is given to campus administrators 
only rather than to all partici! In .a hierarchical’ organizational structure, 
campus administrators are resopeibh le for integrating the objtctives of their ¢ 
; organizational units with the objectives of the institution. They are responsible, 
also, for allocating resources i. harmony with established priorities. Ccrpus 
administrators, therefore, are the locyg of decision-making regarding objectives 
and priorities., In traditional delphijsall participants are chosen on the basis 
‘of their expertise, hence all participants are the locus of decision-making on 
objectives and priorities. In contrast to another iteration of written partici- 
pant response and feedback, the next step in the modified delphi is to paleo 

dctions with campus administrators in a face-to-face discussion. 


Maier aided that decision-making involves quality and acceptance dimensions, 
Quality refers to how well a decision squares with objective facts. Acceptance 
refers to the degree to which the group that must execute the decision accepts 
it. In Maier's view, high quality and high acceptance are both needed ‘for effec- 
‘ tive decisions. (1963, pp. 252 and 253). 


Considerable research has demonstrated the superiority of group over individual 
decisions and has fostered the use of "management teams" in "participative 
dgeision-mak ing" (e.g. Buttenmiller (1972) and Leveille (1972). 


Research has demonstrated, also, that change is more acceptable when the people 
affected by it have helped to create it (e-g. Pollack, 1971). Indeed, in an 
experimental test of nominal, delphi and interacting decision-making processes, 
Van De Ven (1972) found that face-to-face grouns perceived greater: satisfaction 
from,their participation than did delphi participants. 


Hence » Planning actions in a face-to-face meeting with campus administrators should 
lead to more effective decisions than would another round of written hecktetgarten 
by all administrators. : . 


The final ‘step in both traditional and modified delphi is to circulate a final 
written report to all participants. This step concludes the process by informing 
participants of the outcomes of the process. 


Summary 


In- terms: of administrators’ time, cost, effort required and outcomes, a modified 
delphi technique which alternates face-to-face with written interaction to provide 
both rational and emotional outlets, provides an efficient and effective method 
for obtaining institutional consensus on research needs and priorities. 
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7 , 
7 . . A Computational Example 
' V $e eft of 
How to ‘Calculate ImpOxtgnce/Consensus Rankings 
. ( - : 
5 
yn - . 
ra a ot : ve 
* A. Calculate Importance Ratings : D. Rank Consensus Scofes , 
(Arithmetic Mean of Item Responses) ~ . * Consensus - Consensus 
: t , , Need Scores Ranking 
e.g, Responses T 5.2 4 
: 5 2 5.6 5 
3 3 3.0 2 
4 ‘ 4 2.4 R 1 ‘ 
5 5 3.8 "3 
. 7 2 
: , Sample 4 4 
19 + S'= 3.8 Importance - 
. Rating 


B. Calculate Consensus Scores E. Sum Importance + Fopassie Ranks 


(Sum of Absolute Values of each ™ Importance ~- Consensus Sum of 
Respondent's choice - Importance Rating) Need Rank Rank Ranks 
; 4 
iS Importance Absolute ; / , : : 
e.g, Choice | Rating = _ Value_ 5 A 2 5 
5 3.8 1.2 4 1 1 2 
3 3.8 ¢ 8 5 5 3 8 
4 3.8 a2. , 
a a 3.8 1.2 
? a 3.8 *1.8 
i Sample 
5.2 = Consensus 
, 5 . Score ¢ 
C, Rank Impor~ance Ratings sat F. Re-Rank the Sum of Ranks . 
' .t “Importance Importance Sum of —Importancé/Consensus 
ee Need Ratings: Rankings Need Ranks Ranks 
as ie ae oT aes 
3.8 4 1 8 4.5 
2 2.6 2 2 7 3 
. 3° 3.0 3 3 so” 2 
4 1.4 1 4 2 1 
5 4.2 5 5 8 4.5 
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